Skip to content

Add Local Dashboard Manager for Profilers #2000

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 20 commits into from
Nov 6, 2025
Merged

Add Local Dashboard Manager for Profilers #2000

merged 20 commits into from
Nov 6, 2025

Conversation

@goodwillpunning
Copy link
Contributor

@goodwillpunning goodwillpunning commented Sep 8, 2025 •
edited
Loading

Changes

What does this PR do?

  • Adds initial implementation for uploading an AI/BI Dashboard to a "local" Databricks workspace that summarizes the a profiler assessment extraction run.

Relevant implementation details

  • Implemented so that other EDW profilers only need to upload a Dashboard template (JSON representation of the AI/BI dashboard).

Caveats/things to watch out for when reviewing:

  • a Databricks hostname and token are needed to perform integration tests

Linked issues

  • N/A

Functionality

  • added relevant user documentation
  • added new CLI command
  • modified existing command: databricks labs lakebridge ...
  • added skeleton classes
  • ... +add your own

Tests

  • manually tested
  • added unit tests
  • added integration tests

@goodwillpunning goodwillpunning requested a review from a team as a code owner September 8, 2025 21:44
@goodwillpunning goodwillpunning added the feat/profiler Issues related to profilers label Sep 8, 2025
@github-actions
Copy link

github-actions bot commented Sep 8, 2025 •
edited
Loading

✅ 51/51 passed, 10 flaky, 3m4s total

Flaky tests:

  • 🤪 test_validate_invalid_source_tech (172ms)
  • 🤪 test_validate_table_not_found (1ms)
  • 🤪 test_validate_non_empty_tables (11ms)
  • 🤪 test_validate_successful_schema_check (171ms)
  • 🤪 test_validate_invalid_schema_check (1ms)
  • 🤪 test_transpiles_informatica_to_sparksql_non_interactive[False] (13.1s)
  • 🤪 test_transpile_teradata_sql_non_interactive[False] (15.382s)
  • 🤪 test_transpiles_informatica_to_sparksql (14.104s)
  • 🤪 test_transpile_teradata_sql_non_interactive[True] (14.877s)
  • 🤪 test_transpile_teradata_sql (6.531s)

Running from acceptance #2864

sundarshankar89
sundarshankar89 reviewed Oct 6, 2025
View reviewed changes
@goodwillpunning goodwillpunning mentioned this pull request Oct 27, 2025
Merged
7 tasks
@goodwillpunning goodwillpunning force-pushed the feature/add_local_dashboards branch from 2dcdf50 to 25c1724 Compare October 29, 2025 21:22
@codecov
Copy link

codecov bot commented Oct 29, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 52.52525% with 47 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.40%. Comparing base (343e3e8) to head (398dcc9).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...bridge/assessments/dashboards/dashboard_manager.py 52.12% 44 Missing and 1 partial ⚠️
...databricks/labs/lakebridge/contexts/application.py 60.00% 2 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #2000      +/-   ##
==========================================
- Coverage   64.55%   64.40%   -0.15%     
==========================================
  Files          96       97       +1     
  Lines        7953     8052      +99     
  Branches      826      830       +4     
==========================================
+ Hits         5134     5186      +52     
- Misses       2640     2686      +46     
- Partials      179      180       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@goodwillpunning goodwillpunning changed the title [WIP] Add Local Dashboard for Profiler Add Local Dashboard Manager for Profilers Oct 29, 2025
@radhikaathalye-db @goodwillpunning @sundarshankar89
Add method to upload DuckDB files to Unity Catalog Volume with tests (#…
390f9c7 
…2124)

<!-- REMOVE IRRELEVANT COMMENTS BEFORE CREATING A PULL REQUEST -->
## Changes
<!-- Summary of your changes that are easy to understand. Add
screenshots when necessary, they're helpful to illustrate the before and
after state -->
### What does this PR do?
Added functionality for uploading the duck db extract to a UC volume
### Relevant implementation details

### Caveats/things to watch out for when reviewing:

### Linked issues
<!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes,
fixed, resolve, resolves, resolved. See
https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
-->

Resolves #..

### Functionality

- [ ] added relevant user documentation
- [ ] added new CLI command
- [ ] modified existing command: `databricks labs lakebridge ...`
- [ ] ... +add your own

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [ ] manually tested
- [x] added unit tests
- [ ] added integration tests

---------

Co-authored-by: Will Girten <[email protected]>
Co-authored-by: sundarshankar89 <[email protected]>
sundarshankar89
sundarshankar89 requested changes Nov 5, 2025
View reviewed changes
Copy link
Collaborator

@sundarshankar89 sundarshankar89 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small nits: overall it is almost done.

@goodwillpunning goodwillpunning self-assigned this Nov 5, 2025
sundarshankar89
sundarshankar89 approved these changes Nov 6, 2025
View reviewed changes
Copy link
Collaborator

@sundarshankar89 sundarshankar89 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

gueniai
gueniai approved these changes Nov 6, 2025
View reviewed changes
Copy link
Collaborator

@gueniai gueniai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@goodwillpunning goodwillpunning added this pull request to the merge queue Nov 6, 2025
Merged via the queue into main with commit 8d61193 Nov 6, 2025
9 of 10 checks passed
@goodwillpunning goodwillpunning deleted the feature/add_local_dashboards branch November 6, 2025 16:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gueniai gueniai gueniai approved these changes

@sundarshankar89 sundarshankar89 sundarshankar89 approved these changes

@radhikaathalye-db radhikaathalye-db Awaiting requested review from radhikaathalye-db

Assignees

@goodwillpunning goodwillpunning

Labels

feat/profiler Issues related to profilers

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@goodwillpunning @gueniai @sundarshankar89 @radhikaathalye-db